The software development process often involves testing. The purpose of testing is to confirm that an application is functioning as intended. Software and/or hardware used to perform tests on functional code may be referred to as a “testing application.”
For example, Continuous Delivery (CD) is a software development model that aims to shorten the time period between (a) the writing of updates and/or improvements to a particular software product and (b) the delivery of the updates and/or improvements to consumers of the particular software product. Many companies employing a CD model utilize a deployment pipeline and/or a delivery pipeline. The pipeline includes a set of stages through which each piece of software must pass on its way to release. The pipeline may be automated such that when one stage completes, the next stage automatically begins. If all stages are completed, the piece of software is automatically delivered to the consumers. One or more stages of the CD pipeline may include one or more testing applications, used to evaluate functional code of the software product. If a testing application does not find any problems with the functional code being tested, then the next stage of the pipeline may be initiated. The updates and/or improvements of the software product may be delivered to consumers of the particular software product. If the testing application finds a problem with the functional code, then the testing application may generate an alert. In response to the alert, a software engineer may debug and/or modify the functional code. The revised functional code may be placed into the pipeline again for testing, delivery, and/or deployment.
However, testing applications may themselves be unstable or otherwise not function as intended. In other words, it may not be possible to be confident that a testing application is producing test results that supply meaningful information about the functional code being tested by the testing application. A testing application may fail for various reasons, such as errors in the testing application's code or problems in the environment in which the testing application is running. For example, the testing environment may experience network congestion, a hardware failure, or some other type of failure not related to the functional code being tested. Accordingly, the fact that a testing application fails may not provide helpful information about the functional code being tested.
The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Therefore, unless otherwise indicated, it should not be assumed that any of the approaches described in this section qualify as prior art merely by virtue of their inclusion in this section.